In a computer system with a distributed memory configuration (for example, a cluster), each computing node has direct access to its own attached local memory. The memory attached to other nodes is called remote memory. Usually remote memory is not directly accessible and message passing mechanisms are used to communicate between the nodes.
Remote Memory Access (RMA) refers to a software interface that gives the impression that the remote memory can be directly accessed. There are many forms of RMA operations, but ultimately they can be summarized as two: read and write. The read operation copies data from a remote memory address range into a local buffer. The write operation copies data from a local buffer into a remote memory address range.
Existing Remote Memory Access (RMA) implementations generally fall into two categories. The implementations corresponding to the first category make use of the Remote Direct Memory Access (RDMA) capability of the underlying interconnection fabrics between computer nodes InfiniBand (IB) Host Controller Adaptors (HCAs) such as those from Mellanox support RDMA functions in hardware. The capability is exposed to applications via a software interface called IB Verbs. With IB verbs, in order to perform an RMA operation, the user creates a work request and posts it to a work queue. The HCA then processes the queue and performs RDMA in hardware. The approach used by the second category is to emulate RMA functions over a regular message passing interface (MPI). An example is the implementation of one-sided operations in MPICH/MPICH2 (High-Performance MPI). This is often done in a request-reply fashion. Due to the asynchronous nature of the RMA operations, a separate thread is usually needed to ensure the requests get processed.
The existing RMA implementations have numerous drawbacks. Under RDMA-based implementations, specialized hardware is required. Under message passing-based implementations, additional traffic pertaining to associated messages must be transferred over the interconnect fabric, thus reducing the effective bandwidth of the fabric.